Hey @elastic/kibana-platform,

Would you mind giving this PR a spin? There is no need to do a code review (unless you really want to give preliminary feedback!) as I'll split it into more reviewable chunks/PRs later on.

Also there are no recommendations on how to test this one since I'm really interested in the way you use CLI on day to day basis so that we can notice issues I've missed.

Please, see Known issues or changes in behaviour and let me know what you think should be handled or improved on the very first iteration. There are likely some other subtle differences I haven't mentioned, so feel free to comment about any suspicious CLI behavior.

Thanks!

Okay, here's quick update on the issue above:

• 500 issue was happening since core was forwarding request to the "legacy" Kibana/Hapi that was triggering route handlers and request lifecycle hooks that were registered before optimization is started and Elasticsearch plugin is initialized ( TypeError: Cannot read property 'getCluster' of undefined for server.plugins.elasticsearch.getCluster('admin')). This could happen when you tried to access Kibana in non-dev mode or bypass base path proxy in dev mode (using :5603 directly) during optimization phase. In dev mode base path proxy would have properly "paused" all requests.

• With "legacy" Kibana you'd just get Unable to connect since "legacy" Hapi isn't listening on port yet.

• In the core we can either "pause" requests until Kibana is ready similar to what base path proxy does or just reject requests that would be similar to "Unable to connect" we have in master. I don't really want to add "pausing" logic overhead in request proxiyng code as it's called for every request, so I decided to just respond with 503 and Retry-After header (and Kibana server is not ready yet body - may not be the best wording though) until Kibana is fully initialized because it feels like Service Unavailable is what really happening here.

I've updated first point in Known issues or changes in behaviour with these details as well.

Does that make sense to you @spalger @epixa?

I'd probably convert them to main CLI sub-commands e.g. ./bin/kibana keystore create

The CLI commands are consistent across the stack, so we don't want to change them unless there's a good reason and we coordinated that with the ingest and elasticsearch teams.

I do think it's important that we don't log the host and port until all endpoints are actually loaded up. It's nice that the base path proxy handles it for you, but I'm far more interested in the behaviors when the basepath proxy is not present than when it is. I just tested on a snapshot build of this branch and it took multiple seconds between when we logged the statement and when Kibana would stop spitting out errors.

This doesn't appear to have anything to do with optimization because it's happening on a fresh install from a snapshot, so the optimizer is not firing.

Perhaps we could just pass a function with the binding logic into legacy server that it can invoke when it's all loaded up?

The CLI commands are consistent across the stack, so we don't want to change them unless there's a good reason and we coordinated that with the ingest and elasticsearch teams.

I got you, but it seems it doesn't stop us from using ./bin/kibana keystore and ./bin/kibana plugin for ./bin/kibana-keystore and ./bin/kibana-plugin under the hood, right? Anyway, it's out of scope for this task, but I'll keep that in mind.

I do think it's important that we don't log the host and port until all endpoints are actually loaded up. It's nice that the base path proxy handles it for you, but I'm far more interested in the behaviors when the basepath proxy is not present than when it is. I just tested on a snapshot build of this branch and it took multiple seconds between when we logged the statement and when Kibana would stop spitting out errors.
This doesn't appear to have anything to do with optimization because it's happening on a fresh install from a snapshot, so the optimizer is not firing.

Yeah, optimizer is just the slowest thing, but we also have a bunch of other initialization logic in the legacy world.

Perhaps we could just pass a function with the binding logic into legacy server that it can invoke when it's all loaded up?

We don't need to pass anything since core now manages kbn server and core/.../legacy_service can just log this message whenever kbn server is ready (additionally I'll downgrade messages emitted by core/.../http_service to debug so that it doesn't distract users, but will still allow us to get more insight into core life cycle during debugging).

I got you, but it seems it doesn't stop us from using ./bin/kibana keystore and ./bin/kibana plugin for ./bin/kibana-keystore and ./bin/kibana-plugin under the hood, right? Anyway, it's out of scope for this task, but I'll keep that in mind.

I'm not really sure what you mean by this, so let's just sync up whenever we do tackle that.

We don't need to pass anything since core now manages kbn server and core/.../legacy_service can just log this message whenever kbn server is ready (additionally I'll downgrade messages emitted by core/.../http_service to debug so that it doesn't distract users, but will still allow us to get more insight into core life cycle during debugging).

Awesome!

I tested running Kibana with a snapshot build as well as installing a plugin with a snapshot build, and other than the feedback I had about the timing of the hostname/port log, it seemed good.

I'm sure I'll have more concrete feedback on a PR by PR basis.

As it turned out we can make core responsible for the bootstrapping of Kibana even without new platform CLI (see #22190), so I'm closing this one for now.

/cc @elastic/kibana-operations, just in case you may find something useful in this PR at some point since, if I understand correctly, CLI is on your plate.